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FOREWORD 


The U.S. Department of Labor is sponsoring a series of subject-specific train-the-trainer 
workshops in support of the implementation of the JTPA Amendments of 1992. These 
workshops represent the initial offerings of the capacity building network as authorized by 
the Amendments. In most cases, the Guides (TAGS) and training materials that form the 
foundation for each workshop are based on research conducted by training contractors, 
under DOL sponsorship. The workshop training activities present key findings of the 
research that can be applied to the JTPA system and reinforce practical skills related to the 
TAGs. 


This Practitioner’s Guide on Assessment for At-Risk Youth serves as a companion piece 
to the Tool Kit, also prepared by Brandeis. This publication is one in a series of guides that 
has been developed to complement turnaround training at the state and local level. In 
addition to this workshop, other workshop topics in the series include: 


» Assessment: Issues addressed include information necessary to design client 
assessment and individual Service Strategies. Also discussed are self-evaluation of 
assessment practices, alternative organizational structures for assessment 
components, and selection of appropriate assessment methods and instruments. 


» Case Management: Issues discussed include case management at the client and 
systems levels. 


» SDA Monitoring of Service Providers: Issues discussed include procedures and 
instruments for use by SDAs in monitoring contractors or service providers. 
Emphasis is on program quality for nine service activities and compliance 
requirements for general financial management and payroll. 


» Targeting, Outreach and Recruitment: Issues discussed include effective tools for 
planning, evaluating and conducting targeting, outreach and recruitment activities at 
the SDA level. Also discussed are the best practice methods and techniques, names 
of knowledgeable persons and clarifications of related portions of the Act and 
regulations. 


» On-The Job Training (OJT): Issues discussed include developing quality OJT 
programs that are competency driven, have measurable outcomes, and meet the 


longterm needs of the participant. Also addressed are implications of the 
amendments on the OJT program design. 


» State Oversight: Training for State staff responsible for JTPA oversight is based on 
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two technical assistance guides. One guide concerns program compliance 
requirements and the other focuses on monitoring SDAs program quality. 


The training manuals and summaries, in conjunction with the Trainer’s Primer, provide a 
basis for developing training tailored to the needs of the State and SDA personnel. Users 
are encouraged to adapt the materials presented and incorporate their experience and their 
expertise into subsequent turnaround training provided at the State and local level. 
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INTRODUCTION 
WHY ANOTHER GUIDE ON ASSESSMENT? 


Employability assessment for at-risk youth has drawn iegislative and programmatic attention 
over the last several years. Many papers, speeches, and headlines have heralded progress 
as well as setbacks on this subject. Why then is it necessary to prepare not just another 
Practitioner’s Guide on this topic but a comprehensive Trainer’s Tool Kit and Decision 
Maker’s Summary as well? 


Isn’t it common knowledge that assessment is a critical component of quality and cost- 
effective programming in employment and training systems? Every experienced practitioner 
knows this, and many have cultivated an assessment vision or philosophy at the local level. 
However, it seems that many other practitioners (and, more importantly, many of the young 
people in JTPA programs) are still not gaining all or even some of the benefits of a 
comprehensive, integrated, quality assessment system. In the course of our research, and 
in the process of working with more than 100 service delivery areas on these issues, we 
uncovered many reasons assessment systems are not being developed, implemented, and 
operated in such a way that they contribute to quality and cost-effective programming. 
Some of these reasons are technical, some political. 


Many youth practitioners have reported that they understand the importance of assessment 
and the basic components of assessment, but have been frustrated repeatedly in attempts 
to face the above challenges and put the "best practices" into use. This Practitioner’s Guide 
is designed fur them. Consequently, the Guide (and its companion Decision Maker’s 
Summary and Trainer’s Tool Kit) goes beyond the essential components of a 
comprehensive, integrated, quality assessment system and begins to answer these important 
questions: 


» Why isn’t knowledge about assessment being translated into the formulation 
of quality systems for at-risk youth? 


» What are the current challenges around assessment? 

» What strategies can lead to the effective development, implementation, and 
operation of quality assessment systems which include all the essential 
components and gain all the benefits? 

» What is keeping practitioners from using assessment strategies effectively? 


» And how can practitioners manage so as to promote and maintain quality in 
their assessinent systems? 
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The Practitioner’s Guide is organized in five parts: 


Section One sets the context. It suggests a working definition of assessment 
for at-risk youth, presents the reasons why assessment is an important part of 
developing effective programming for at-risk youth, summarizes the current 
status of youth assessment systems, and provides an overview of the legislative 
context for assessment. 


Section Two highlights the underlying principles that guide the development 
of a meaningful strategy for at-risk youth. While these principles are not new, 
they are underutilized. Our purpose in reviewing them here is to challenge 
the system to change old thinking and old habits so as to serve the current 
generation of at-risk youth effectively. 


Section Three offers strategies for putting the assessment principles to work. 


Section Four examines what gets in the way of implementing really good 
employability assessment systems. 


Section Five presents a management strategy that will ensure that the quality 
of the assessment process is maintained. This section recommends adapting 
a "Managing for Quality" strategy for the purposes of quality control and 
quality improve.nent of assessment systems. 
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© SECTION ONE 
SETTING THE CONTEXT 


DEFINING TODAY’S AT-RISK YOUTH 


Assessmc-nt strategies must be context-specific and youth-centered. That is, they need to 
be designed and implemented based on a solid understanding of the program's target youth 
population. The process involved in carefully defining the population to be served ensures 
that communities fully understand and build consensus around allocating limited resources 
and designing appropriate programs. 


Using Group Characteristics. The traditional approach to defining today’s at-risk youth is 
to use demographic characteristics. Practitioners have long known that youth who share 
certain demographic factors -- such as low-income status, racial minority, school dropout, 
teen parent, welfare recipient -- are at high risk of long-term unemployment. Many 
communities therefore use these or other demographic characteristics to define “at-risk 
youth" and then design services to meet the needs of youth who meet that profile. 


However, using group characteristics as the sole determinant of who is at risk has several 

igni t drawbacks. First, group characteristics are really proxies for more specific skill 
deficits and social problems. They are not easily compared to employer requirements in 
defining employability, nor are they readily translated into needs that can guide a service 
strategy. As such, their value as a basis for individual assessment and community planning 
is limited. Second, using group characteristics as a basis for targeting means that some 
youth are likely to be arbitrarily defined as "at risk" when their skill levels do not justify that 
definition. Without information that is more specific than demographic characteristics, 
practitioners are limited in their ability to make quality choices about allocating resources. 


Using a Skills-Based Approach. As employers increasingly emphasize employability skills 
as the primary determinant in their hiring decisions, practitioners are i 
responding by defining youth at-risk based on deficits in or level of, employability skills, 
especially functional basic skills and/or pre-employment/work maturity skills. 


A skiils-based approach lets practitionezs define "at-riskness" in relation to the employability 
stills required by local employers; those most at risk of long-term dependency are those 
with the lowest level of employability skills. Another advantage of defining youth at risk 
based on skill level is that it enables practitioners to assess the different needs within the 
at-risk population very specifically, e.g., individuals testing above or below specified reading 
and math levels. This lets program operators develop appropriate curricula and program 
designs for meeting different levels of need within the population, and allows targeting and 
planning decisions to be made with more precision. 
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Despite its advantages, however, a skills-based approach fails to take into account the group 
characteristics that have proven to be social or cultural barriers to employment. 


Using a Mix of Skills and Characteristics. By combining skill measures and group 
characteristics, practitioners can develop a “hybrid” definition of at-risk youth that includes, 
for example, those who are dropouts, or minorities, or teen parents, and who lack specific 
educational and/or work skills. A hybrid definition lets practitioners target youth with 
clearly specified employment skills needs while also formally recognizing some of the social 
and cultural factors that exacerbate the risks of failure in the labor market. This approach 
may help JTPA and other youth-serving agencies to develop common definitions and a 


common approach to assessing participants. 


DEFINING ASSESSMENT 


Onre you have defined at-risk youth, the first step in developing, implementing, and 
operating an effective assessment system is reaching agreement on a working definition of 
assessment. As defined in Curnan and Fiala’s “Assessing Employability for Results 
(National Governor’s Association, 1986): 


“Assessment is the on-going process of determining an individual client's 
strengths and skill deficits relative to job requirements and employer 
expectations, selecting an appropriate service strategy; and measuring the 
client's progress in skills acquisition using explicit and formal criteria." 


Since assessment is an ongoing process, it is important te lock at when, how, and why 
various assessment activities occur as the youth moves from program entry to program exit. 
Assessment information must be used for more than intake procedures and eligibility 
determination. Rather, assessment goes beyond those two functions to include a host of 
activities that result in ongoing youth-centered employability planning and accountability. 
An ongoing, comprehensive assessment process has these components: 


» Appraisal of a youth’s level of employability, including capabilities, interests, and 
other characteristics, so as to determine eligibility and make initial service 
assignments. 


» In-depth assessment, to diagnose each individual youth's needs using a variety of 
tools and techniques, to provide feedback on results of diagnosis, and to develop an 
appropriate employability development plan (EDP) that matches services to the 

youth’s individual needs. 
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» Monitoring or benchmarking progress through ongoing assessment, thereby creating 
a series of “wins” (interim successes) for the youth. Information collected and 
development plans based on new information, and enables them to provide feedback 
and counseling based on assessment results. 


» Documenting and certifying the achievement of goals. Post-assessment results 
provide information on whether the youth is ready for employment. 


As we stated earlier, employability 

programming. Youth employment and training programs cannot work towards employment 
outcomes or determine which training and services are necessary to achieve social and 
economic self-sufficiency unless there is some process in place for deterniining youth needs 
relative to labor market requirements. This information also becomes the basis for 
documenting individual progress and certifying individual accomplishments. 


ity assessment is the key to quality and cost-effective 


At the program level, assessment information is the main component of employability 
development planning, which lays out the steps to be taken along a path to self-sufficiency. 
Assessment information contributes to employability development planning by: 

» helping match services to individual client needs, 

» pre-riding the base for developing exit criteria that ensure that individuals 


have achieved proficiency levels (i.c., competency) for the different basic 
academic, work maturity, and job skills being taught; and 


» helping practitioners update ED Ps with the new information gained through 


interim assessment and monitoring. 


At the systems level, assessment provides information to help increase the effectiveness and 
efficiency of job training services. Assessment information helps answer questions such as: 


» Who do we want to serve? 
» What outcomes do we want to achieve? 
. What training and service needs have to be met? 


in short, when it works well, assessraent is the method for dealing with the complex array 
of s vices and organizing them tc match youth needs. 
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The following chart defines assessment at different levels of the employment and training 6 
system: 


a ———— 

©On-going method of defining goals and 
determining where the youth is relative 
to goals 

®Method of defining outcomes based on 
employer expeciations. 

eInformatior gathered and interpreted 
to make decisions about eligibility, 
program assignment, benchmarking and 
progress and certification 
@Nondiscriminatory method for 
selecting/hiring 

@Pased on skills anc abilities 

©Process for answering policy level 
questions: Who do you want to serve? =| ® 


What outcomes do you want to achieve? 
What services do you need to get there? 


¢ Use aggregate assessment data to 
evaluate the effectiveness and efficiency 
of training services. 


At the national level, there has been a strong legislative push underscoring the importance 
of assessment since the Job Training Partnership Act was enacted in 1982. The current 
legislative environment not only encourages, but in many cases mandates, the development, 
implementation, and operation of quality assessment systems. 


JTPA. A basic assumption of the original JTPA legislation was the necessity to review and 
document through assessment in order to determine what services an individual should 
receive. The 1988 Job Training Partnership Advisory Committee recommended the use of 
more in-depth diagnostic assessments to improve quality of services, and called for the 
development of more effective and sophisticated assessment systems for serving 
disadvantaged populations with serious skill deficits and multiple needs. The JTPA 


amendments require that each participant 9 
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be assessed as the basis for an individually tailored service plan. The amendments call for 
the use of assessment systems to: identify and select participants; determine and verify 
eligibility, identify participant skill leveis and service needs; specify the competency levels 
to be achieved and serviced to be provided; and evaluate individuals’ progress toward 


JOBS. The Social Security Act requires states to operate a Job Opportunities and Basic 
Skills (JOBS) program. Initial assessment of cach participant is required regarding 
educational and supportive service needs, skills, prior work experience, and 

levels. Assessment results must be used to create an Employability Development Plan 
(EDP). However, beyond this individual-level assessment, JOBS requires states to conduct 
a systems-level assessment as a way to determine future service demands, improve targeting 
efficiency, and assure that participants receive tr::.1¢ ‘hat is linked to labor market 
demands and results in positive outcomes. 


Vocational Education. The Vocational Educatior and Applied Technology Act of 1990, 
which amended the Perkins Vocational Education Act, expanded the state’s responsibility 
to develop and implement performance standards and measures, to perform initial state 
assessments and annual program evaluations, and to use the resulting data in the state plan. 


Literacy. The National Literacy Act of 1991 emphasizes the need for assessment. The Act 
established a National Institute for Literacy and requires it to conduct basic and 
research and demonstrations on the assessment of literacy skills and the development of 
instructional techniques, and to include assessment tools in a national database. Grants 
fund naticnal strategies to determine appropriate assessments of workers’ literacy and basic 
skill needs. States must use their basic adult education grant to develop and implement 
quality indicators that are consistent with JTPA, Perkins, and JOBS standards. 


The challenge for practitioners and policy makers alike is to support improved content for 
instructional effectiveness, rather than simply more assessment for compliance purposes. 
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© SECTION TWO 
NINE GUIDING PRINCIPLES FOR ASSESSMENT 


Practitioners from around the country have convinced us that ..e key operational points 
made in Section One of this Prectitioner’s Guide and elaborated upon in assessment 
literature, are "being missed," if only by degrees in some places. In practice, the importance 
of content has given way to a focus on structure. Findings from the recent National 
Evaluation of Youth Employment Competency Systems (YEC) support this observation: 


Greater emphasis needs to be placed on the content rather than the form of YEC 
programs. Monitoring efforts have focused primarily on whether YEC programs are 
consistent with the form of a sufficiently developed system. SDAs have, by in large, 
conformed to requirements; the framework for YEC programs is in place in most 
SDAs. What is required now is more attention to the content of thoze programs.' 


Put differently, there seems to be a blurring of the line between “compliance testing” and 


the field’s general desire to improve learning opportunities for youth through adequate and 
authentic “instructional assessments." 


Programs must clearly identify why they are assessing. The rationale for assessment 
generally falls into three categories: 


1. | Tobe in compliance -- with the law and the regulations only. 


2. To be accountable -- to the clients, to the program itself, and to the 
community the program serves. Using assessment for this purpose lets 
programs document results based on individual young people's attainments 
relative to their individual needs. 


3. To develop employability -- to use assessment information to move individual 
youth through a continuum of services based on their needs and goals. 


However, compliance alone is an inadequate basis for assessment. All programs seeking 
quality improvements need to assess foz accountability and employability development 
purposes. Assessment systems that are developed to promote accountability and 
employability development will easiiy be in compliance with the law and regulations. The 
reverse approach -- developing systems specifically for compliance purposes -- may or may 
not result in systems which have value to the learner. Accountability is a good thing when 
the focus for design is on being accountable to the primary customer -- the at-risk youth. 
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The principles below summarize the key elemeats of exemplary assessment systems that 
promote instructional effectiveness: 


1. 


sis 


ate SSESSMENE SYSCEMS VIEPi' if norougr StaNnGcif a Vino 38 Dei 
assessed and what they are being assessed for. All staff engaged in assessing at-risk 
youth must understand the characteristics of adolescence. Likewise, they need to 
eee — a ee 


a, reasoning, attitude, and other elements that define 


intermittent events. Assessnient is interwoven throughout intake, skill diagnosis, 
service planning, progress monitoring, and outcome. Staff involved in all these 
program components need to be trained in assessment (at various levels) and their 
observations need to be sought and used. 


: e cesses -- measuring or 
collecting iafonmetion, — the —_s making judgments about the 
information, and assigning consequences based on the information. All of these 
subprocesses must be in place in order for assessment to be done in a quality way. 
Practitioners usually describe their assessment sequence as a series of steps, each 
with a discrete beginning and end; in reality, a quality assessment process is a 
sequence of subprocesses that overlap in time and space. Staff need to realize that 
assessment is a continuous thread winding through all other program components. 


| ery — the best tools — — — are cummed ——— The 
name of a test is not always a good indication of its intended use. For example, a 
"reading test" that has been designed to measure a young person's vocabulary is only 
valid for that specific purpose; it will not be effective for judging reading 


comprehension. 


0) bgram_ design and should gather only 
information which hes a ri 7. — and can be addressed by ihe program. 
Assessments in youth employment programs should help policymakers identify their 
target population, their desired outcomes, and the mix of services necessary to 
achieve those outcomes. Assessment measurements in youth employment programs 
must relate to employer requirements and expectations. 


labor market skill reoulrements Thus, employability assessment must be 
individualized, incorporating a variety of tools and techniques that provide multiple 
opportunities for youth to apply their skills in a work-relevant context. 
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f intensity. This is the only way assessment 
can — instructional —- Too often, i in employment and training, we 
expect one test or assessment system to serve both compliance and instructional 
improveinent purposes. For assessment to promote instructional improvement, the 
assessment process must identify each individual youth’s existing skill levels and skill 
zaps so thzt we know exactly what is needed in order to bring that youth’s abilities 
up to the labor market’s minimum employability level. Each individual may need 
intensive, mid-level, or minimal intervention; the level of intensity of the ntervention 
will be different for each individual, and even for the same individual as related to 
different skill and knowledge areas. 


All staff can be part of the assessment process. Because quality assessment is an 
ongoing process that permeates all components of a youth employment program, all 
staff -- from receptionist to counselor -- should be trained (at varying levels) to assess 
employability. This investment in staff training is critical: it provides multiple 
opportunities for youth to demonstrate their skills, helps maintain quality control and 
cost control, and acknowledges that the assessment process begins the moment the 
youth walks in the door. 


lor ens at t nt system serve: 101 The case manger 
makes sure that assessment data are utilized throughout the employability 
development planning process and that the young person understands how each type 
of assessment relates to the various components of his/her service plan. 
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SECTION THREE 


IMPLEMENTATION STRATEGIES 
FOR PUTTING PRINCIPLES TO WORK 


Seasoned practitioners have found the following strategies effective in implementing 
quality assessment systems for at-risk youth: 


Create a new paradigm for thinking about assessment. 
Directly assess skills relevant to the labor market. 
Provide multiple assessment opportunities for youth. 


Connect assessment to instruction and worksite training by changing 
curriculum so what you teach is what you test. 


Measure competence within the context of employability readiness, not 
classroom learning. 


Develop and implement a cohesive, well-thought out staff training and 
development plan to support the assessment system. 


Practitioners need to start with a strong philosophical base about assessment to develop 
an effective assessment system. Not only must they understand the nine guiding 
principles of assessment that are discussed in Section Two, but they must also advocate 
for them, teach them to others, and work to institutionalize them. 


Practitioners who struggle with the daily challenges of operating a program often find it 
easy to think of assessment as "only a test." This thinking suggests they need only to 
purchase an off-the-shelf test and administer it as a pre-test and post-test in order to 
have a quality assessment system that is easy and cost-effective. In a recent CHR 
practitioner survey, the questions practitioners most frequently asked about assessment 
were: 


» What test should I use? 
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» How can I incorporate these new ideas about assessment into the system 
we currently have in place? 


The prevalence of these questions shows the need for a dramatic shift in thinking. The 
reality is that no single test can do it all. And, significant changes often need to be 
made to current systems in order to incorporate the principles of assessment. 


However, although assessment should permeate every component of an employment and 
training program, assessment is only cue of the issues that practitioners must deal with 
on a daily basis. Therefore, development and implementation of a quality assessment 
system requires that every staff member and decision maker understand and support the 
guiding principles of assessment so that the subprocesses that comprise the assessment 
process -- collecting information, evaluating information, judging information, and 
assigning consequences based on information -- become an ingrained part of the 


program. 


Thinking about assessment as a thread running through all the other program 
components provides a new way to emphasize that all staff members need to get 
involved in collecting assessment data. Each staff member has responsibility for some 
part of the four subprocesses that make up the assessment process. An important part 
of this new paradigm is that the sequence of subprocesses that make up the assessment 
process are not linear; rather, they overlap and build on each other. 


This new paradigm of assessment as an ongoing information collection process means 
that, at various points, staff take a "time out" and use the information they have 
collected to make a judgment. The type of judgment that is appropriate at any given 
time -- appraisal, diagnosis, benchmark, or certification -- depends on the quality and 
quantity of information available, and the implication of the judgment (that is, the 
consequence assigned as a result of the judgment). 


2. DIRECTLY ASSESS SKILLS RELEVANT TO THE LABOR MARKET. 


No assessment system can be effective without a clear, agreed upon definition of 
employability. Assessment is the process by which to measure an individual’s strengths 
and skill deficits relative to job requirements, employer expectations, and individual 
goals; therefore, any assessment system needs to start with an understanding of job 
requirements, employer expectations, and the individual youth’s goals. 


Rapid technological change, innovation, and heightened competition have changed the 
American workplace. These workplace changes have changed the skills employees need 
in order to achieve long-term success in the workforce. American companies are moving 
away from the "Taylor" work environment in which a few educated planners and 
supervisors oversee large numbers of line workers performing simple, repetitive tasks. 
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ASSESSMENT SEQUENCE 


CONSEQUENCE EVALUATION 


JUDGEMENT 


MEASURE > JUDGEMENT-----> CONSEQUENCES 


Center for Human Resources 
Brandeis University 
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This methcd -- well-suited for mass production at inexpensive prices -- is not effective at 


achieving the high quality products and services, variety, and responsiveness to change 
that are needed to compete in today’s global marketplace. 


The high performance work organization is replacing the Taylor work environment. In 
this model, front-line workers have increasing levels of responsibility, this results in 
increases in flexibility, quality, and productivity. High performance work organizations 
require new business strategies such as: collaboration, exemplary customer service, total 
quality management, and participatory management. 


What does this mean in terms of employability skills? 


The key to new work environments is a highly trained workforce made up of workers 
who possess a broader set of skills. Employees are increasingly expected to serve as 
decision makers at the point of production or sale. 


The traditional basic skills -- "the 3 Rs," reading, writing, and arithmetic -- are no longer 
sufficient for workplace success. As the information processing demands of most 
occupations have grown, basic employability skills have evolved to include higher-level 
thinking skills, the capacity to learn new tasks, and the ability to solve increasingly 
complex problems. The Committee for Economic Development noted several years ago 
that "mastery of the old basics of reading, writing, and arithmetic may be sufficient for 
entry-level jobs, but because of the constantly changing nature of work, minimum skills 
are not sufficient preparation for career advancement. Schools must make a greater 
effort to develop higher level skills, such as problem solving, reasoning, and learning 
ability." 


Work maturity expectations have also changed over time and grown more sophisticated. 
At one level, the change is minimal -- studies of employer hiring requirements still tend 
to focus on the basic work maturity elements -- punctuality, regular attendance, positive 
attitudes and behaviors, and the like. But as with the other basic employability skills, the 
definition of work maturity is also changing in important ways. 


As employers look at new ways of organizing the workplace, they are increasingly 
emphasizing more sophisticated social skills as essential for even entry-level employees. 
The American Society for Training and Development has included personal 
management, group effectiveness, and leadership skills among the skills necessary for 
success in the labor market.’ The recent report of the Secretary’s Commission on the 
Acquisition of Necessary Skills called for what it termed "workplace know-how" and 
placed heavy emphasis on a series of sophisticated "interpersonal skills" as one of the 
core sets of competencies required by the high performance work environment.‘ Those 


skills included teamwork, teaching others, serving clients/customers, exercising 
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leadership, negotiating, and working with cultural diversity. Even relatively unskilled 
workers must have the interpersonal skills necessary to deal with co-workers and 
customers. 


America is looking for a new kind of worker, one who possesses higher-order thinking 
skills which include problem-solving skills, listening skills, negotiation skills, creativity, 
and the ability to set goals, This is occurring at a time when more and more of the 
clients for employment and training programs enter our doors with fewer and fewer 
skills. In short, employers want employees who know how to learn and adapt to the 
changing workplace. To meet these enhanced employer expectations, the employment 
and training system needs to address and upgrade assessment and competency systems 
and begin to "retool" to provide more intensive, longer term training. 
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In defining employability skills, 


— — i to start with employers. Local definitions of employability, and the 
assessment systems built around them, are useful only if area employers concur. 
Employer validation (more than just PIC approval) is essential. 


Practitioners identify three approaches to employer validation: employer-specific, 
occupation-specific, and entry-levei-specific. All involve direct contact with area 


and all involve analyzing local jobs and their skill requirements. They 


differ in the degree of analysis of individual employer contact and/or occupational 
specificity that is involved. Employer · aid occupation-specific approaches provide 
opportunities to build relationships with local businesses and tailor a system to their 
needs, but at a higher cost in terms of time and effort. 


Employer-specific validation involves job task analysis for each entry-level 
position in a specific firm to identify the basic employability skills required to 
perform each task. When tabulated, the information indicates the basic 
employability skills required for entry-level employment with that particular 
employer. ‘When repeated with other employers within the community, the 
process leads to a profile of a “generic worker" in the local labor market. 


ion focuses on job requirements in a 


specific — (e. g-, brick- aver, office worker, etc.) All employers who 
hire that type of worker identify the requirements for specific entry-level 

positions; this information is used to define entry-level requirements for that 
occupation. Those requirements are then tabulated with those in other occu- 


pations, yielding a set of generic employability skills. 


Entry-level-specific employer validation begins with many employers 

identifying the basic employability skills required for any entry-level 

employment. Through surveys and focus groups, employers identify those 

skills they consider important and explain how their employees use those | 
skills. The resulting information is used to identify a series of functional basic 
employability skills used by the generic entry-level employee. 


Regardless of approach, employer validation is an important part of the process of 
creating an employability assessment system. It is critical to work with the business 
community to decide what skills to assess and what proficiency levels are necessary to 
achieve competency in specific skill areas. 
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Different Measurement Strategies. As they rethink the basic skills and work skills that 
youth and adults need to succeed in the workforce, practitioners and policymakers are 
taking a fresh look at how those skills should be a-sessed. In the past year alone, two 
major reports -- the National Commission on Testing and Public Policy’s From 
Gatekeeper to Gateway, and the Commission on the Skills of the American Workforce’s 
America’s Choice -- have questioned the value and use of traditional, norm-referenced, 
multiple choice tests alone as accurate indicators of individual skills and achievement. 
Rather, they argue that a combination of indicators is needed to provide a complete 
picture of achievement. Noting the need for fairer, more realistic methods of measuring 
ability, both Commissions have argued for use of more direct, “authentic” forms cf 
assessment. 


Among practitioners, there has been a similar movement towards what is often called 
“alternative assessment" -- that is, assessment through means other than paper-and- 
pencil, norm-referenced tests. The development of criterion-referenced assessment tools 
is a direct result of the interest of employment practitioners in focusing assessment on 
measuring specific, workplace-related competencies and the individual's ability to apply 
those competencies in the kinds of situations they are likely to encounter in real life. 


Many practitioners are working on the cutting edge in this area. One alternative being 
widely explored is performance-based assessment. Performance-based assessment is the 
direct observation and review of the application of skills against set criteria. Using 
portfolios of student work, demonstrations, group projects, and other activities, 
performance-based assessment directly examines how individuals use their knowledge 
and skills in realistic situations, rather than measuring skills indirectly through some 
form of written multiple choice test. 


Once again, careful use of language is important in the serious exploration of alternative 
assessment approaches. "Performance assessment" and "authentic assessment" are often 
used interchangeably, however, they are not the same. "Performance assessment" refers 
to the types of responses you wish to elicit from the learner (e.g., a behavior, a product). 
"Authentic assessment" refers to the context in which the learner responds.’ For 
example, our current systems always impose time restrictions on testing. To make a 
performance assessment authentic (i.e., to relate it to the world of work), one must 
relate the time constraint to the skill, behavior, knowledge being demonstrated. Rather 
than debating whether or not tests should be timed, we need to ask whether a real 
worker, using the skill being tested, would be timed as s/he used that skill on the job; if 
so, we need to determine what a realistic time constraint for ‘ic behavior being tested 
would be in a work setting, and then we need to use tl, : constraint in the test. 


It may be that all authentic assessment is performanc _.ed, but not all performance- 
based assessment is authentic unless it is designed to be so. 
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The idea of performance assessment is not new -- it has long been used in the arts, 
athletics, and even employment and training (the mock job interview is a prime 
example). What is new is the growing interest among educators and others in using 
performance-based assessment to better measure a broad range of basic skills, including 
basic skills. 


There are a number of different types of performance-based assessment. These include 
portfolios, open-ended questions, demonstrations, and interviews. What ties the various 
performance assessment strategies together is the emphasis on providing a more 
accurate and “authentic” assessment through the actual demonstration of skills. The aim 
of performance assessment is not only to test what a student knows, but whether the 
student can use his or her knowledge and skills appropriately. (See the boxes on pages 


ahead for a description of each approach.) 
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PERFORMANCE ASSESSMENT APPROACHES 


Performances, Demonstrations, and Exhibitions. At the heart of performance 
assessment is the idea of demonstrating or exhibiting mastery - showing what you can 
Go. Performances and demonstrations allow participants to do just that: 
demonstrate what they have learned. The most familiar form of “perfromance” for 
youth practitioners may be the mock job interview or worksite evaluations. A youth's 
performance at the interview or at the workplace during a set period of time is 
evaluated according to clearly defined criteria. But demonstrations can also be used 
to assess a variety of functional basic skills - for eaxmple, by requiring students to 
read an instruction manual and then perform the correspinding activity. This type of 
exercise allows students to demonstrate their ability to read, comprehend and apply 
information. As the emphasis on the ability to use basic skills grows, some form of 
demonstration becomes an essential element in assessment. 


Portfolios. Portfolio assessmcnt brings together a number of examples of a student's 
work as a means of documenting his or her progress or achievementsover time in one 
or more areas. A portfolio might contain a mix of materials such as research notes, 
drawings, essays and journal entries, and often includes drafts and revisions as well as 
final "best" peices. Portfolios are particularly valuable in assessing writing skills, but 
they could also be used in evaluating a participant's career decision making or job 
search skills or work readiness. One of the major benefits of the portfolio approach 
is that it actively involves students in reviewing and making judgements about their 
own work as they determine the framework for the portfolio, select the contents, and 
judge its quality. Portfolios also help students and instructors assess the learning 
process, by demonstrating improvement over time or the differences between draft 
and final products. As such, portfolios serve two purposes - providing evidence or 
achievement while also engaging the student in the assessment process. 


Group projects... 
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— > dies veut subetiviqnen, pasteclacly f cetpesed to ctandasdieed, ccmm- 
referenced, multiple-choice tests: 


in, a ee ape a 
examine a person's ability to apply skilis ir context. When properly designed, 
performance-based assessments accurately reflect real-life educational and job 

in contrast to many norm-referenced testis which are administered in 
isolation and focus on simple, short, written, fill-in-the-blank and multiple-choice 
questions. (A word of caution, however: performance-based assessments can be 
as inauthentically designed as traditional tools, and thus can be equally 
irrelevant.) 


ills, skills that 


have | been identified consistently by a as — = ’s workplace. 
For example, they require students to demonstrate judgment and interpretation in 


addressing a question, skills increasingly necessary even in entry-level jobs. 


process. Jo ested cualles, tailing to too elite dene to lenmness iactend of wih 
learners; learners are not informed about the reason for the assessment, results 
are not shared, and no effort is made to work with the learner to use the 
assessment results in planning a program. In performance assessment, the learner 
is an active participant in the assessment process; in many cases, the learner 
actually designs the specific assessment task with the instructor. 


— —— if — developed, — — ee 
focus more clearly on what a person can do and allow flexibility for various 
learning styles. For example, norm-referenced tests are generally administered in 
one standard time period which rewards speed of recall and penalizes slow 
answering. A person who answers more slowly will receive a lower score. This 
can lead to inaccurate inferences about a person’s ability. In this instance, a low 
test score does not necessarily imply that a person does not possess the 
knowledge or skills being tested -- only that s/he works more slowly. 


peers and supervisors. — — — is as 3 po 
developing process skills as it is about content. It is important for students to 
develop their ability to work well in group situation since most adult and 
workplace challenges will require the capacity to balance individual and group 
achievement. 
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Rethinking the Assessment Process. Performance assessment offers a number of 
important advantages in assessing the kinds of employability skills increasingly requized, 
but it is no more an automatic “solution” to the assessment question than is selection of 
a specific test. Instead, discussions of performance assessment should help remind us 
that the assessment process is more than simply choosing a test. The “best” test in any 
given situation depends on what is being taught, which depends on how employability 
has been defined locally. 


Good tests (and mediocre ones as well) can be easily misused. The emphasis must 
be on what is being measured (i.c., the definition of employability) and how the 
information will be used (for instructional and/or for compliance purposes). Some 
examples of how good tests can be misused follow: 


e Using a locator or appraisal test (¢.g., the TABE locator, the CASAS 
appraisal, the ABLE locator) to monitor progress by using it to 
measure gains. While these locators are "good" tests, they help meet 
compliance needs; they do not diagnose where learning begins, nor do 
they measure progress. 


Using the Job Corps Reading Test as a measure of reading 
comprehension. Again, it is a good test, but it measures vocabulary 
more than reading comprehension. Nor should it be used beyond 
compliance purposes, it does not diagnose where learning begins nor 
does it measure progress. 


Using the Wide Range Achievement Test (WRAT-R) to inform 
instruction and measure progress. While the WRAT-P. is a useful tool, 
the reading portion measures vocabulary as opposed to reading 
comprehension. While it is a fairly comprehensive measure of math 
competence, it does not fully assist instructors in planning an 
instructional approach to attack skill deficiencies. 


To assess employability skills appropriately, practitioners need to look at assessment as 
an ongoing process that examines a variety of skills at a variety of times using a variety 
of different instruments and methods. The methods of assessment must relate to the 
skills you want to measure; those skills must have been validated by the labor market; 
and the information generated must be used in ways that zeinforce the learning process. 
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Assessment information must be linked to the labor market, the instructional process, 
The initial design connection is to the labor market. Assessment strategies and systems 
should not be developed in a vacuum; they must be based on the labor market, so that 
they measure relevant labor market skills and so that judgments about an individual's 


abilities will be made within the context of labor market needs as well as the individual's 
needs and desires. (This is discussed in Strategy #2, earlier.) 


Assessment information must also be linked to instruction, so that the instructional 
process relates to the individual's proficiencies and deficiencies. This means that: 


» Information must be collected in a way that informs the instructional 
process. Competency-based (or criterion-referenced) tools and strategies 
lend themselves readily to the instructional process because those tools and 
strategies assess specific skills and identify the learner's level of difficulty in 
mastering the skill. 


» Systems should be designed so that they move relevant assessment 
information to the instructional setting (classroom or worksite). 


» Teachers/worksite supervisors must be involved as part of the total 
employability development process. 


» Curriculum-embedded assessments should be created so that the learner 
perceives them as a learning activity and the teacher uses them as an 
opportunity to measure and evaluate the learner's abilities. 


In sum, this strategy addresses assessing for what you teach and teaching toward what 
you assess. The first step in making this a reality is to understand what needs to be 
taught and how it will be taught. Then assessment tools and strategies can be identified 
and developed which will measure skills that are relevant to the instructional setting. 


The previous strategies have addressed the philosophy and subprocesses of assessment: 
» how we think about assessment overall; 
» how we identify what the labor market needs as a threshold; 
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» how we use that labor market information to select appropriate tools and 
strategies, 


» why it is necessary to use various types of tools and strategies to make 
accurate judgments; 


» how we use assessment information to inform a=d imrpove the quality of 


This strategy addresses one more subprocess that is necessary in order for the 
assessment proccss to be authentic -- how to make learning and assessment both 
contextual and integrated. 


Contextual learning (and assessment) means shifting from traditional school-based 
learning approaches to ones that use knowledges and skills most often found at work. 
To paraphrase Lauren Resnick from the University of Pittsburgh, contextual learning 
focuses on cooperation rather than individual cognition; contextual learning focuses on 
the use of tools rather than on pure thought; and contextual learning focuses on the 
manipulation of objects and events rather than on abstract symbols. 


Integrated learning involves teaching (and testing) the knowledge of a subject’s concepts 
and principles simultaneously with the application of those concepts and princivles. At 
work, knowledge and skills (i.e., the application of knowledge) are most often practiced 
together; therefore, both our instructional and our testing mechanisms must find ways to 
teach and measure them together. The nature of skills differs depending on the context 
within which they are being applied and the content to which they are being applied. 


At the point of mastery certification or final certification of overall competence, it is 
necessary to assess an individual's ability to use knowledge and skills in job-related 
settings. This can only happen if we use measurement strategies which take context and 
integration into account. 


As a job training agency begins to define and assess employability, staff development and 
training become key priorities. Staff training builds staff understanding, involvement, 
and support for the system: the more staff understand how an employability assessment 
system can help them and their clients, the more likely they are to use and take seriously 
the tools and procedures that are available. 


On a more technical level, staff training on using the assessment tools is essential to 
maintaining the validity and reliability of the assessment process, particularly for 
performance assessment which involves observation and significant professional 
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judgment. However, roles and responsibilities regarding the assessment process will vary 
among staff positions. A receptionist may have responsibility only for observing and 
reporting specific behavior and an intake worker will probably collect and record data, 
while a case manager will be responsible for judging assessment daia so as to make 
appropriate service assignmenis. The receptionist must be trained to observe, while the 
case manager must be trained to evaluate. 


Practitioners identify several elements as essential to an effective staff training and 
development process: 


» Training must be broad-based. All staff must acquire a broader mix of skills, 
including technical, vision, and political skills in order to deal with the challenges 
of implementing a quality assessment system. All levels of staff within the agency 
must be trained, because the assessment process affects everyone fron. counselors 
and trainers to intake workers, planners and managers, subcontractor aff, and 
worksite supervisors. Perhaps more important, a comprehensive assessment 
system requires training across agency lines. Joint staff training is a particularly 
effective means of building bridges among local agencies and ensuring that 
"ownership" extends to staff throughout the system. Joint staff training also 
reduces redundancy in information-gathering across institutions, staff will only 
trust assessment information from other institutions if they feel confident that the 
staff in those institutions are well qualified -- a direct result of training. 


» Training and development should be ongoing, and not just a one-time occurrence. 
Program staff need time to work their way through the implications of a new 
system and to respond to problems as they arise. As one practitioner pointed out, 
“it’s not until you start working with the system that you start to see the things 
you thought you understood, that you don’t; to see the nuances you didn’t 
anticipate." 


» Training must be competency-based. As with participant training, competency- 
based staff training offers the advantages of clearly defining the skills that need to 
be learned, concentrating training on the specific skills that each staff person 
needs to develop, and emphasizing the ability to use needed skills appropriately as 
the standard for success. 


Finally, a comprehensive staff training and development plan does not stop at formal 
training (e.g., seminars and workshops). It also includes the development of quality 
criteria which are used to observe staff as they carry out different tasks and processes. 
Using quality criteria for staff « servation provides an opportunity for immediate and 
continuous feedback to staff on how to improve their skills. 
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9 SECTION FOUR 
SELF-EVALUATION FOR EFFECTIVE ASSESSMENT 


Even when practitioners commit to the assessment principles and use proven strategies 
in developing, implementing, and operating quality assessment systems, the results are 

not always as positive as one would hope. To remedy that, we need to understand how 
to address issues that may inhibit the potential effectiveness of the proven strategies. 


Issues that affect the development, implementation, and operation of a comprehensive, 
integrated assessment system tend to fall into two categories: issues with.n the direct 
control of system decision makers, and issues external to the job training system. This is 
complicated by the fact that the assessment system can be affected by a mix of both 
technical and political issues, any of which can "come at" the system from internal or 
external sources. 


THE IMPORTANCE OF INTERNAL DECISION MAKERS 


Decision makers’ actions and beliefs affect the staif's ability to develop, implement, and 
operate a quality assessment system. Decision makers determine the time, money, and 
commitment available to support an assessment system; the decision makers’ own 
philosophy, education, and training are factors as well. If decision makers think of 
assessment as “just a test,” individual staff members will have a hard time championing a 
more comprehensive strategy. 


Decision makers also determine how much time and resources to allocate to assessment. 
If decision makers do not place a high priority on assessment or lack a strong 
philosophical base, they may not understand that assessment requires an investment of 
time and money, just as do curriculum and instruction. 


The staff's ability to develop, implement, and operate a comprehensive, integrated 
assessment system depends on the answers to questions such as these: 


» Do decision makers understand the importance of staff training and do 
they invest accordingly? 


» Are decision makers flexible and willing to adapt to new and changing 
information, or do they respond to suggested changes with, "We've always 
done it this way, so there is no reason to change"? 


» How committed are decision makers to developing an effective assessment 
system, and how involved are they in the process of implementing and 
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operating one? Do they offer their input? 9 


» Can they motivate staff despite external obstacles and hassles? Do they 
refuse to say, "It can’t be done"? 


The following matrix can help determine the extent to which decision makers may be 
acting as obstacles to effective assessment: 
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PAGE 


@We need to do assessment 
because of regulations 


o Assessment is important but 
we need to spend more time 
on curriculum and instruction 


©The staff can handle it just fine 


©] keep tabs on the situation 
@Staff send me periodic 


reports 
el get involved if there is a 


Staff Training 


Staff are trained in how to use 
assessment forms 


Flexibility/ 


©@We've always done it this way 


©] am willing to make changes 
within the context of our 


current system 
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Assessment is a separate 
component 


@Our assessment staff interpret 
data and this information is 
linked to curriculum and 
instruction 
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DECISION-MAKERS’ ASSESSMENT HINDRANCE MATRIX 
RESULTS: 


6-12: 
Very High Hindrance Level -- Your assessment system is basically a pre-test upon which to make eligibility decisions and a 
post-test to document positive terminations. You are not gaining any of the benefits of a comprehensive, integrated 
assessment system. 


13-18: 

High to Medium Hindrance Level -- Although you have a philosophy base, for whatever reason (not investing the time or 
resources, not investing in staff training, being inflexible to change, etc.) you have not taken this philosophy and turned it 
into action. 


19-24: 
; i -- You have a strong philosophy base and are committed to assessment. You have 


passed this commitment and philosophy onto your staff. You have instituted many of the important elements of a 
comprehensive, integrated assessment system; however, there are still barriers and issues you are working through. It is 
likely that with the necessary commitment, you can go all the way. 


25-30: 

Very Low Hindrance Level -- You have institutionalized all of the critical elements of a comprehensive, integrated 
assessment system. You and your staff have turned a strong commitment and philosophy base into action. Assessment is 
integrated into every aspect of program design, and you are not hindering the process. Congratulations! 
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support from internal decision makers is only half the battle. External factors, 
ranging from turf concerns to confidentiality issues, may hinder the assessment 
ao Factors that affect assessment can be technical or political. 

that makes it possible to resolve both technical and political issues 

—— Go aatenees eatin & Gin ae 
youth-serving agencies. This can be addressed through tactics such as interagency task 
forces, interagency meetings, cross-training, and regular communication to identify and 
address issues in a timely and mutually acceptable way. 


Resolving technical issues will require a thorough understanding of the technicalities 
involved so as to devise a plan of action that effectively addressed the issue. For 
example, many programs find it difficult to consolidate information-gathering because of 

ality laws, To address this problem, the decision maker needs a good working 
knowledge of the state confidentiality laws so that s/he knows what can and cannot be 
done in terms of having information follow the client through the system. 


The first of the political issues is the conflict between quality and compliance that 
underlies the entire assessment discussion. Pre/post testing at entry and exit is sufficient, 
if compliance is the only purpose of assessment. However, a quality assessment system 
that improves instructional quality and provides accountability to the learners for 
ultimate outcomes requires that assessment be integrated into the classroom and at the 
worksite. Even once practitioners in the youth employment system are committed to 
focusing the assessment system on increasing instructional effectiveness (i.e., quality), 
they may need to convince those in other youth-serving agencies that compliance cannot 
be allowed to drive the design of the assessment system. A balance needs to be struck 
between accountability for the quality of an assessment system and accountability for 
compliance purposes, and practitioners within the JTPA system need to lead the way in 
the broader community. 


As with any systems change, there is likely to be significant resistance to change. 
Resistance to change may be inevitable, but it can be reduced by taking the time to 
make sure everyone is on board, and encouraging sharing and input into the 
development of the system. The time, hard work, headaches, and hassles of instituting a 
comprehensive, integrated, quality assessment system make everyone’s job harder. But, 
if everyone understands how assessment can improve program outcomes and make their 
own work more productive, and if they are involved in developing the system, a sense of 
ownership will help reduce anxiety about systems change. 


Another implementation issue at the political level is the challenge of integrating 
assessment into program design. Because integration requires people to change the way 
they do their work and the way they think, the need to integrate assessment into 
program design is inevitably problematic until all staff members have been trained in 
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assessment and until assessment has permeated every point of the youth employment 
process. Practitioners need to think about how the young client’s assessed needs drive 
every program design decision. Assessment should be a central factor in decisions about 
changes in curriculum, instruction, or other program design issues, and vice versa. How 
should assessment results influence program design? Will program design changes 
require modifications to the assessment system? The solution is to educate both staff 
and decision makers so everyone involves feels a sense of ownership of the system and 
are willing to undertake the advocacy and hard work necessary to implement it. 


Like resistance to change within organizations, practitioners are likely to face turf 
concerns both within and across agencies. Practitioners need to develop a working 
environment that encourages open communication and sharing within and across 
agencies. This can be achieved through cross-departmental meetings and cross- 
functional problem-solving teams, in which staff can build relationships and learn how 
different departments fit together to make the assessment process possible. Decision 
makers need to take the lead role in cross-agency collaboration by negotiating stronger 
relationships with their peers in other agencies. Inter-agency meetings or teams, as well 
as intensive education and information-sharing, are also effective techniques. 


Among the technical issues that hinder implementation, resource constraints are always 
pivotal. Policymakers need to decide how to balance enhanced or expanded assessment 
against available resources. This will likely involve building on what is already in place, 
reallocating resources, pooling resources, and/or seeking additional resources. 


Time constraints are related to resource constraints. If decision makers are misinformed 
about how much time assessment requires or are insensitive to conflicting demands on 
staff time, they may fail to allocate enough time to the assessment function. This 
problem requires open and extensive communication between decision makers and staff 
about how to structure a realistic timetable and what kinds of compromises are 


acceptable. 


Confidentiality may be a difficult or even an impossible snag. Confidentiality rules, 
which differ by jurisdiction, affect the circumstances in which a participant's records may 
be accessed. Confidentiality concerns may cause wasteful duplication of information 
collection. Decision makers and staff need to clearly understand the relevant 
confidentiality issues and work on an inter-agency basis to develop practical strategies. 


Legal issues need always to be kept in mind. The assessment process in job training 
programs is not exempt from the general litigiousness of American society. However, 
for the most part, new legal concerns are merely a stronger incentive to utilize 
assessment in an ethical way. 
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For example, it’s the right thing to do to administer tests in such a way that they 

reflect the abilities of people with disabilities. Now, the Americans with 
Disabilities Act has codified that ethical framework in legislation, specifying that if a test 
is used in an employment setting, it cannot be used to screen out persons with 
disabilities unless the criteria used to develop the test are the same as those required to 
perform: the job and are consistent with business necessity (Section 102(b)(6) of the 
Americans with Disabilities Act of 1990). 


Likewise, using race-normed assessments is not the right thing to do; now, for 
programis are specifically prohibited from race-norming the GATB (General Aptitude 
Test Battery). 


The most far-reaching legal consideration involves the liability for assessment, and again 
this is a back-up for an ethical framework for assessment. One of the component 


subprocesses of assessment is making judgments, and these judgments have 
consequences. — ee oe 


ny fieet, was the decision making process o conssnmal ene (1... did the 
youth agree with the consequence assigned as a result of the assessment judgment?); and 
second, was the intent to broaden rather than narrow the youth's ultimate employment 
options? 

There are a lot of new legal considerations that can create legal problems related to 


assessment. The best defense against legal problems is understanding the relevant legal 
issues and implementing the assessment system within an ethical framework. 
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To improve the quality of results, the quality of the processes that yield those results 
have to be improved. Although this seems a simple enough concept, the private sector 
has been exploring its implications for the past several years. 


In a recent survey, for the first time, businesses in the United States ranked quality, not 
profits, as their top goal. In developing quality, businesses are adhering to a 
management philosophy known as Total Quality Management, initially developed by 


Fifty-seven percent of American companies have Total Quality Management as a 
strategic goal or policy, according to a recent American Society for Training and 
Development survey, and the remaining 43 percent anticipate adopting Total Quality 
Management within the next one to three years. Since the public sector job training 
system is totally dependent upon private sector placements for its ultimate success, 
public sector agencies would be well advised to join private companies in ensuring a 


focus on quality. 


Making quality the number one goal for the public sector will require a dramatic change 
on the part of agency leadership as well as an examination of external environments 
which impose certain inflexible processes and practices on local organizations. The 
recent emphasis on increased accountability, especially within JTPA, has created an 
intense focus on “compliance.” A "managing for quality" philosophy requires that quality 
not be compromised to compliance. 


"Managing for quality" means attempting to change an organization’s culture by 
managing people's efforts rather than managing people. It means changing the way 
people do things by focusing on process improvements.* However, managing for quality 
is more than simply identifying an organization's processes, establishing teams to develop 
improvement strategies, and controlling the variations. 


Managing for quality requires visionary leadership and a belief system that surrounds, 
encompasses and permeates discreet activities, teamwork and process evaluations. In 
short, managing for quality is hard work, it requires day-to-day involvement of leaders, 
and it is never “over.” Instead, it is an ongoing way of thinking, doing, and believing. 
Managing for quality can help practitioners and policymakers alike shift the paradigm for 
assessment, as callec for earlier in this Guide. 
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This chapter is not intended to serve as a comprehensive guide to implementing a 
"managing for quality" system. Rather, it takes some of the principles of the private 
sector’s Total Quality Management philosophy and explores how to apply them to 
managing for quality in one process -- employability assessment for at-risk youth. 


The basic tenets of A “managing for quality’ philosophy can help set the context for a 
discussion on improving the quality of the assessment process. Some employment and 
training have managing for quality over the past few years, largely under 
the direction of the Center for Remediation Design and the Center for Human 
Resources. Their experiences have proven that it is important to understand and 
commit to a set of principles, strategies, and tools before undertaking a managing for 
quality approach. 

Much has been written about Total Quality Management, with each author setting his or 
presented here are all of Edward Deming’s work; Implementing Total Quality 


Management: An Overview, by Joseph R. Jablonski; and The Fifth Discipline: The Art 
and Practice of the Learning Organization, by Peter Senge. 


Six principles that define the fundamental values of Total Quality Management are 
essential for managing for quality in public sector organizations: 


1. A focus on the customer (in our case, the "customer" is at-risk youth); 


2. A focus on the process as well as the results (the assessment process and 
outcomes); 


3. A focus on prevention over inspection (information collection and 
evaluation on an on-going basis rather than at a single point in time; e.g., 
analyzing what went wrong when a youth drops out and changing the 
program to prevent that outcome from reoccurring); 


4. Mobilizing the expertise of the workforce (line staff are given authority and 
responsibility to identify the criteria against which a process will be 
evaluated for quality and are empowered to change the process based on 
study); 


5. Fact-based versus assumption-based decision-making (processes are 


changed based on facts collected against criteria rather than based on 
whim or gut feelings); 
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6. Feedback (reg: lar recognition of accomplishment, on-going examination of 
progress toward mutual goals for both customers and workers). 


In order to act on the above principles, strategies have to be identified and then 
implemented. Strategies that are central to implementing a “managing for quality” 
philosc phy include: 


» Participatory management -- at all levels, from line worker to agency 
director. It evolves over time and is based on trust and feedback. 


» Continuous process improvement -- with the goal of defining quality and 
never allowing for variations or defects. Continuous process improvement 
demands constant improvements versus occasional reviews. 


» Teams -- sstablished as cross-functional, both vertically and horizontally. 
Teams are xesronsible for understanding a particular process, establishing 
process contrcis, making decisions, and implementing improvements. 

Some of the tools for implementing a “managing for quality" philosophy include: 

» Brainstorming skills 

» Facilitation skills 


» Group dynamic/team member skills 


» Process control (identification of process flow and process variation) 


» Staff training 


While your organization may not be changing its philosophical approach to overall 
management overnight, the basic principles, strategies, and tools presented here can be 


used to improve the quality of any given process and/or subprocess. 


While Total Quality Management was developed in the private sector and most case 

studies involve manufacturing, the service sector and particularly the public service , 
sector have many processes which can be examined through a “managing for quality’ 

lens, including: 
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» the employability development process; 

» the assessment process; 

» the orientation/eligibility/intake process, and 

» the basic skills instructional process. 
Clearly, many of these processes overlap. The key is which one is the focus of 
improvement -- the others may become subprocesses to the process under study, as it 
may become a subprocess to another. 


The question before us is: 


It is important to define three terms as we begin this discussion: 


Process -- A series of operations or activities linked together to provide a 
result that has increased value. 


Quality -- Those attributes of a product or service to which the custo;ner (in 
our case, at-risk youth) attaches value. 


Criterion -- A standard, rule, or test on which a judgment can be based. 


Let’s expand on each "managing for quality" principle, and examine it through the 
window of the assessment process. As discussed in Section Three, assessment can be 
broken down into four subprocesses: collecting information, evaluating information, 
judging information, and assigning consequences based on the information. These 
subprocesses are each overlaid on the principles and criteria which follow. 


Principle #1. Maintain a F he C | 


Customers must be asked regularly what they want and what they value about what the 
service is or should be. But first, the customer has to be identified. In the employment 
and training context, is the primary customer of the assessment process the state, the 
client, employers, other agencies, etc” From our perspective, the primary customer is 
the individual youth participant. As such, ihey are the ones who define quality; 
consequently, all decision-making must be youth-centered. By focusing on the individual 
youth, we guarantee that we are designing and delivering services that focus on where 
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that individual youth is on the continuum toward employability. 


Identifying the primary customer is a key exercise because too many businesses articulate 
one customer but set up their practices to respond to a totally different customer. For 


example, they say the customer is the buyer, yet they but set up practices which respond 
to the supplier. 

Understanding the characteristics of the primary customer is another important exercise. 
Employment and training systems cannot respond to the needs and wants of at-risk 


youth unless they have a solid understanding of the characteristics of today’s at-risk 
youth, as discussed in Section One earlier. 

Employment and training systems provide many examples of this mismatch as well. For 
example, when asked directly, programs often identify at-risk youth as their primary 
customer; however, if the assessment systems focus on compliance, that is an indication 
that the real, or hidden, customer is the funding agency. This example illustrates the 
concern for balancing accountability for the quality of an assessment system and 
accountability for compliance purposes. 


What standards could be specified and then observed to ensure that a youth focus is maintained? 
Potential Criteria for Assessment Process: 
The characteristics of at-risk youth are well understood. 
Assessment results are valued by these youth. 
Assessment results produce a quality and quantity of ideas for action. 
The quality of staff judgments i⸗ validated with the client. 
Staff allow the client to make choices from multiple options. 


Staff provide flexibility in the process to neei ongoing individual needs. 


Managing for quality is a zero-defects philosophy. Quality is continuously improved 
regardless of how high the quality may be. Customers’ unmet expectations of a product 
or service serve as indicators that something is wrong with the process that produced 
them. Continuous improvements require a structured approach to problem-solving. 
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What standards could be specified and then observed to ensure a focus on the process as well as the 
results? 


There are a variety of tools, techniques, and strategies to collect information 
(measure) about the youth’s level of employability. 


Information that is constantly collected, constantly evaluated, and constantly judged is 
recorded as part of the progress record. 


All staff who interact regularly with the youth meet as a team regularly. 
Observations of youth behavior are recorded by all staff using established criteria. 
Information follows youth among all agencies over time. 


Workers ask youth regularly about what they expect from the process. 


Managing for quality moves us away from the notion of "inspecting quality in." The 
inspection (a manufacturing term which can translate to the public systems monitoring 
for compliance function) means examining the product or service at the end, after 
resources have been spent, for defects and variation. Once defects are found, the 
general response is to provide more inspectors at progress points along the way. 
Adherence to this principle means constant evaluation of the process so as to improve it, 
not more inspections along the same process. 


What standards could be specified and then observed to ensure a focus on prevention rather than on 
inspection for its own sake? 


Youth are asked regularly about what they thought of the services they received. 


Information about individual young people is collected and evaluated on an ongoing 
and regular basis. 


Measurement strategies are developed using labor market criteria. 
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This means increasing the level of employee involvement in problem identification, 
solution development, decision making on what solutions will work best, and 
implementation. This principle, probably more than any other, requires the greatest 
change among and between managers and workers. Workers can no longer come up 
with ideas and then go to management so that management can unilaterally say, "this is 
a good one, this is a bad one." Rather, workers through teams have the power to 
implement their ideas. The organizational culture permits testing of ideas, worker 
responsibility, and decision making. 


What standards could be specified and thea observed to ensare that the expertiee of the workforce ls 
being mobilized? 


Potential Criteria for A P 


All staff involved with youth meet regularly as a team® to evaluate new assessment 
information and to validate/revise/change judgments and consequences. 


Decision making about the youth is controlled by the team and the youth. 
The organization has a strong commitment to staff training. 


* The team identified here is a youth (client) team (made up of the youth and the staff involved with 
that specific youth’s plan) not necessarily a staff process team (a ‘eam specifically set up to deal with 

the assessment process). If the whole organization were developing a “managing for q zality” vision, an 
assessment process team would also be in place as well (with members of youth teams) whose function 
was to study the total process and make continuous improvements. 


Principle #5. Make Decisions Based on F 


Managing for quality means developing a structured approach to problem-solving. 
Understanding defects in the service and variations in the process and brainstorming 
solutions sets an environment for always creating opportunities for improvement rather 
than opportunities for "got ya." Commitment to a "managing for quality" philosophy 
means creating mechanisms which collect and record factual data about the process 
upon which decisions for improvement are based. 
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What standards could be specified and then observed to ensure that 


Interpretation of the assessment results are specified in relation to the youth's 
employment goal, not in relation to the goal of program activities. 


Measurement strategies are varied and include authentic perform.ance-based 
assessments. 


Youth are observed against certain behavior criteria from the time they walk in the 
door (like being on time, oral interaction) to provide more information to evaluate 
when taking an initial judgment. 


Principle #6. Seek Feedback. 


This is the linchpin of managing for quality. Feedback can be provided in many forms: 
team members — ———— about the process improvements; flow charts 


feedback i in —— toa set criteria and technical assistance is immediately 
provided. 
What standards could be specified and then observed to ensure continuous feedback? 


Potential Criteria for A P 
° Youth receive regular and ongoing feedback on assessment information collection. 


Youth participaie in evaluating the information and forming judgments. 


Staff regularly ask youth for feedback on the process. 


As you may have noted, many of the criteria overlap, helping to define quality for more 
than one of the principles. These criteria, or those that you develop, should be 
consolidated into one set of quality criteria. The quality criteria set the standard for 
how quality for this process is defined. 


WHY DEFINE QUALITY? 


The preceding section was used to discuss each of the "managing for quality" principles 
an¢ to identify standards which can help apply each principle on a practical level. 
Fundamental beliefs can be believed but not practiced. Establishing criteria helps us 
"watch" a belief system in action. Criteria assist process designers in evaluating how the 
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beliefs are being exercised. Criteria help address any dissonance between what we say 
we believe as fundamental to how we design and manage systems ard what we actively 
do. 


Remember the question before us: 


How can our assessment process be examined for quality so as to improve the 
quality of the process as part of improving the quality of the results? 
To study the current process, to make improvements, and to develop a strategy for 


continuous improvements, a set of criteria must be established against which to compare. 


We are proposing establishing “quality criteria" using the “managing for quality" 
principles to inform the content of the criteria. 


Clearly, criteria can be established using other contexts. Our main focus is that one 
must have a standard against which to judge something (just as with individual 
assessment where the labor market requirements set the standard for evaluating 
minimum proficiency levels). And managing for quality provides a customer-centered 
(youth-centered) orientation to establishing criteria. Establishing criteria in this way 
assists in carrying out all the "managing for quality" principles within an isolated process 
even though the entire organization may not be consciously managing for quality. 
Finally, establishing standards assists in implementing a continuous improvement strategy 
for one process or many processes. 


The development of quality criteria for key service functions is a tool which helps 
support several of the principles. Quality criteria assists in staying focused on the 
customer by defining the criteria from the customer's perspective. The organization 
must always be asking, "What is it the customer values about this process?" 


Quality criteria are process-oriented. The criteria are established to facilitate critical 
observation of the process. Quality criteria help identify problems and help develop 
process teams to solve the problems and prevent their reoccurrence. Used 
appropriately, quality criteria trigger the development of teams of workers to solve 
problems from the bottom up, rather than relying solely on management to make 
decisions from the top down. 


Establishing criteria wh’ ch define the quality practices within a certain process assists in 
ultimately evaluating that process based on facts, not assumptions. 
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Finally, as one of many “managing for quality” practices, the use of quality criteria serves 
as a technical assistance and training tool. Quality criteria provide ongoing feedback, 
technical assistance, and training to staff on a regular and ongoing basis. 


Developing quality criteria to upgrade practitioner skills and to continuously improve the 
assessment process is not a simple task. It is an evolutionary process of trust and 
feedback which develops over time, rather like the productive relationship we try to 
foster between youth and staff in our programs. It is not an inspection. If quality 
criteria are used in performance reviews, staff have to believe that the criteria are used 
to improve overall practice and to determine what types of professional development 
investments are needed, rather than to determine raises or hiring and firing decisions. 
(Again, like youth, staff need to believe in the value and use of assessment information.) 


Directors have to understand that continuous process improvement means often 
accepting a small, incremental gain as a step in the right direction toward total quality. 
The theory is that substantial gains can be achieved through the accumulation of many 
small improvements which cause a synergy that yields tremendous gains over the long 
term. 


Quality criteria can be used in three fundamental ways to make continuous process 
improvements: 


I. An organization establishes a process team responsible for assessment. 
One function of the team is to make a fact-based decision about the 
quality of the current process. To do this, a team member observes the 
process under study (assessment) for a period of 6-12 months, conducting 
weekly observations of many staff involved in any part of the process. 
Over time, the team identifies trends (from "good" to “needs 
improvement") which establishes the need for changes to the subprocesses, 
the need for worker training, or both. 


2. The quality criteria are used to provide immediate feedback to the worker 
being observed, critiquing the process and making suggestion on how to 
improve. This feedback mechanism is established in an ongoing way, i.e. 
team member observations of the process do not cease when improvements 
have been made, since improvements will always be being made. 


3. Individual workers become observers of their own process. The quality 
criteria are used as a tool for self-evaluation (a key "managing for quality’ 
skill). Every staff person involved in the most minuscule part of the 
process has the quality criteria. Each and every time workers conduct their 
part of the assessment process, they review themselves against the quality 
criteria, conducting a self-appraisal. They then improve the way they do 
their part of the assessment process the very next time they engage in it. 
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Will employment and training organizations that adopt a “managing for quality” 
philosophy for their assessment systems and for other operational elements save money 
as a result? Will a commitment to continuous process improvement through the use of 
quality criteria prepare more young people for employment at lower cost? This remains 
to be seen. 


Logically, the concept of managing for quality should result in cost savings as errors are 
reduced and innovations identified and introduced. What we do know from the private 
sector’s experience with Total Quality Management is that worker satisfaction increases, 
the product or outcome improves, and the quality of the ideas workers generate about 
the product is better. Applying this to employment and training leads us to expect that 
managing for quality in that setting will improve the system’s outcomes -- specifically, 
more youth will be more ready for work once they come through the system. 


The quality criteria, in essence, define quality as what the customer values. They 
articulate what you are striving to become vs. as much as you think you can do. This 
process sets a standard toward which you are always improving. 


Managing for quality is about a structured approach to problem-solving. This is the first 
step in developing a structured approach to: knowing what you want to improve; 
evaluating current practice against quality practice; brainstorming potential solutions to 
improve quality, implementing improvements, studying and evaluating; brainstorming; 


Finally, the use of quality criteria provides a structure for continuous improvements. 
Managing for quality requires that improvement be a daily exercise, not a quarterly or 
annual activity. 
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